On the Entropy of Couplings

This paper studies bivariate distributions with fixed marginals from an information-theoretic perspective. In particular, continuity and related properties of various information measures (Shannon entropy, conditional entropy, mutual information, Rényi entropy) on the set of all such distributions are investigated. The notion of minimum entropy coupling is introduced, and it is shown that it defines a family of (pseudo)metrics on the space of all probability distributions in the same way as the so-called maximal coupling defines the total variation distance. Some basic properties of these pseudometrics are established, in particular their relation to the total variation distance, and a new characterization of the conditional entropy is given. Finally, some natural optimization problems associated with the above information measures are identified and shown to be NP-hard. Their special cases are found to be essentially informationtheoretic restatements of well-known computational problems, such as the SUBSET SUM and PARTITION problems.